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FOREWORD 


The  problem  of  estimating  a  probability  density  function  is  addressed  in  this  report.  The 
algorithm  presented  here  is  the  kernel  estimation  method  using  a  resistive  grid  network  suitable  for 
hardware  implementation.  Results  show  that  the  linear  Resistive  Grid  Kernel  Estimator  (RGKE) 
yields  estimates  that  are  comparable  to  those  formed  using  a  Gaussian  kernel.  It  will  also  be 
shown  that  incorporating  the  inherent  nonlinearities  of  the  RGKE  allows  the  detection  of 
discontinuities  in  the  density  function. 
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ABSTRACT 


The  ability  to  estimate  a  probability  density  function  from  random  data  has  applications  in 
discriminant  analysis  and  pattern  recognition  problems.  A  resistive  grid  kernel  estimator  (RGKE) 
is  described  that  is  suitable  for  hardware  implementation.  The  one-dimensional  linear  RGKE  is 
compared  to  a  kernel  estimate  using  Gaussian  kernels,  and  simulations  are  presented  using  both 
continuous  and  quantized  data.  The  nonlinear  form  of  the  RGKE  is  shown  to  have  desirable 
properties,  such  as  the  ability  to  detect  discontinuities  in  the  density  function. 
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INTRODUCTION 


In  this  report,  the  problem  of  estimating  the  probability  density  function  f(x)  of  a  given 
sample  of  n  real  observations  Xj, ....  Xn  will  be  considered.  The  approach  studied  here  is  the 
kernel  estimator  from  Silverman l  using  a  kernel  that  can  be  implemented  by  a  resistive  grid 
network.2  A  discussion  of  kernel  estimates  and  their  application  to  neural  networks  can  be  found 
in  Specht,3  and  a  review  of  research  concerning  pattern  classification  is  presented  in  Lippmann.4 
It  will  be  shown  qualitatively  that  using  the  linear  resistive  grid  kernel  estimator  (RGKE)  provides 
results  comparable  to  an  estimate  formed  using  a  Gaussian  kernel.  Results  using  the  nonlinear 
RGKE  will  illustrate  the  ability  of  the  resistive  grid  to  detect  discontinuities  in  the  density.  Finally, 
conclusions  and  possible  applications  will  be  discussed. 


CONTINUOUS  RGKE 

The  resistive  network  described  in  Mead2  provides  a  means  of  computing  the  weighted 
average  of  many  input  signals  or  observations.  The  voltage  at  a  node  is  determined  by  the 
weighted  average  of  the  inputs.  Since  the  amplitude  of  the  voltage  due  to  a  single  input  decreases 
exponentially  with  distance,  signals  that  are  farther  away  will  carry  less  weight.  A  schematic  of  the 
one-dimensional  resistive  network  is  shown  in  Figure  1.*  The  equation  for  the  general  univariate 
kernel  estimator  of  constant  window  width  h  is 


*»-=£' i 


where  n  is  the  number  of  data  points,  h  is  a  smoothing  parameter,  and  K  denotes  a  kernel.  For  a 
one-dimensional  continuous  resistive  network,  the  following  equation  can  be  used  as  the  kernel 
centered  at  y 


l*-rl  i 

**<*>•*  L  ■ 

where  L  is  the  characteristic  length,  R  is  the  resistance,  G  is  the  conductance,  and  A  is  a 
normalization  constant.  In  this  case,  the  characteristic  length  L  can  be  identified  with  the  window 
width  h.  The  normalized  voltage  at  each  node  of  the  resistive  network  described  in  Mead2 
determines  the  estimate  of  the  density  function. 

Figure  2  shows  a  comparison  of  kernel  estimates  of  a  normal  density  using  the  continuous 
RGKE  and  Gaussian  kernels  for  various  window  widths.5  It  can  be  seen  from  these  plots  that,  as 
expected,  the  RGKE  yields  a  density  estimate  similar  to  one  using  Gaussian  kernels. 


*  All  figures  arc  included  after  the  References. 
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Since  the  continuous  RGKE  is  a  bounded  Borel  function  satisfying  the  following 
conditions 

J|KRG(x>ldx<00 
JKRG(x)dx  =  1 

IxKrg(x)|  W->°° 

with  h  — »  0  and  nh  — > then  the  estimate/^  (x)  — >  /(x) 
in  probability  as  n  — » ©o.1 


DISCRETE  LINEAR  RGKE 

When  implementing  the  RGKE,  it  is  necessary  to  consider  quantized  inputs.2  The 
probability  density  estimate  for  the  k-th  node  of  a  discrete  network  is  given  by 

hot  = 

1=1 

where  Mi  is  the  number  of  observations  in  bin  i,  A  is  a  normalization  constant,  N  is  the  number  of 
nodes,  and 


y  ~ 


1 


The  expression  for  y  becomes  exact  as  the  number  of  nodes  in  the  discrete  network  approaches 
infinity.  This  equation  is  valid  under  the  assumption  that  linear  superposition  holds. 

To  simulate  the  linear  RGKE,  10,000  data  points  are  drawn  from  a  quantized  normal 
density  with  zero  mean  and  variance  one,  similar  to  that  shown  in  Figure  3.  The  probability 
density  function  is  then  estimated  only  at  the  centers  of  the  bins.  Figure  4  illustrates  the  estimate 
from  a  discrete  linear  resistive  grid  with  a  characteristic  length  of  5.0.  This  shows  that  the  RGKE 
can  yield  a  smooth  density  estimate  of  the  data. 

A  discontinuity  is  introduced  in  the  data  to  evaluate  the  performance  of  the  linear  discrete 
RGKE.  This  is  illustrated  in  Figure  5a,  which  clearly  shows  the  edge  in  the  data.  The  estimates 
from  the  RGKE  for  different  values  of  L  are  shown  in  Figures  5b  through  5d.  As  these  plots 
show,  by  decreasing  L  the  discontinuity  can  be  detected.  However,  smoothness  in  the  estimate  is 
lost  when  the  characteristic  length  is  made  small  enough  to  detect  discontinuities  in  the  data.  This 
fundamental  tradeoff  in  the  choice  of  the  smoothing  parameter  L  is  an  inherent  characteristic  of 
linear  kernel  estimators. 
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DISCRETE  NONLINEAR  RGKE 


The  above  equations  for  continuous  and  linear  discrete  RGKE  allow  for  a  development  of 
the  theory  with  respect  to  kernel  estimators.  However,  including  the  nonlinearities  of  the  network 
in  the  formulation  has  some  benefits.  It  can  be  shown  that  the  following  equations  govern  the 
voltage  at  the  i-th  node  of  a  resistive  grid  network  when  linearity  is  not  assumed: 


GVj 


in 


+  Vfcl 

'  n 


R; 


-1 


G  + 


Ri-i 


.  V|n  =/3Mj 


with 


where  p  is  a  scaling  factor  controlling  the  degree  of  nonlinearity,  V-n  is  the  fixed  input  voltage  to 
the  i-th  node,  and  Ro  is  the  zero  signal  resistance  value.  Again,  the  estimate  of  the  density  function 
is  given  by  the  voltages  at  each  node 


Ji 

N 


2* 


This  set  of  coupled  nonlinear  equations  is  applied  to  the  discontinuous  data  of  the  previous 
section.  Again,  a  histogram  of  the  data  is  shown  in  Figure  6a.  Figures  6b-d  illustrate  the 
probability  density  estimates  for  different  degrees  of  nonlinearity.  A  value  of  0.2  for  P  yields 
results  similar  to  the  linear  case;  the  curve  is  very  smooth  and  no  discontinuity  is  detected  in  the 
density.  However,  as  more  nonlinearity  is  allowed,  the  discontinuity  in  the  density  becomes 
apparent.  With  p  =  1.0,  the  curve  remains  smooth  while  the  discontinuity  in  the  density  function  is 
clearly  shown.  Thus,  the  inherent  nonlinearity  in  a  RGKE  can  actually  improve  its  probability 
density  functional  estimator  qualities. 


CONCLUSIONS  AND  APPLICATIONS 


This  study  has  shown  qualitatively  that  the  RGKE  yields  results  comparable  to  probability 
density  estimates  derived  using  Gaussian  kernels.  The  ability  of  the  nonlinear  RGKE  to  detect 
discontinuities  in  the  density,  while  continuing  to  produce  a  smooth  function,  illustrates  its 
usefulness  as  an  estimator.  Since  the  RGKE  is  suitable  for  hardware  implementation,  it  is  possible 
to  apply  it  to  problems  in  discriminant  analysis  and  pattern  classification  when  real-time  responses 
are  required. 
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FIGURE  2.  COMPARISON  BETWEEN  AN  ESTIMATE  USING  GAUSSIAN 
KERNEL  (SOLID)  AND  RESISTIVE  GRID  KERNEL  (DOTTED) 
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FIGURE  3.  EXAMPLE  OF  DATA  DRAWN  FROM 
NORMAL  DISTRIBUTION,  WHICH  HAS  BEEN  QUANTIZED 
FOR  RESISTIVE  GRID  NETWORK 


FIGURE  4.  COMPARISON  BETWEEN  ESTIMATE  USING 
HISTOGRAM  (DOTTED)  AND  DISCRETE  LINEAR 
RGKE  (SOLID)  WITH  L=5.0 
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BIN  CENTERS  BIN  CENTERS 

FIGURE  5 A.  ESTIMATE  OF  DENSITY  FIGURE  5B.  LINEAR  RGKE  ESTIMATE 


USING  HISTOGRAM  WITH  L=5.0 


BIN  CENTERS  BIN  CENTERS 


FIGURE  5C.  LINEAR  RGKE  ESTIMATE  FIGURE  5D.  LINEAR  RGKE  ESTIMATE 

WITH  L=2.0  WITH  L=1.0 
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BIN  CENTERS  BIN  CENTERS 


FIGURE  6 A.  ESTIMATE  OF  DENSITY  FIGURE  6B.  NONLINEAR  RGKE  ESTIMATE 

USING  HISTOGRAM  WITH  £=0 1 


BIN  CENTERS  BIN  CENTERS 

FIGURE  6C.  NONLINEAR  RGKE  ESTIMATE  FIGURE  6D.  NONLINEAR  RGKE  ESTIMATE 

WITH  0=1.0  WITH  0=2.0 
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